Malignancy assessment for tumors

ABSTRACT

According to a first aspect, there is provided a computer-aided method of malignancy assessment of lesions, the method comprising the steps of: receiving input data; performing a first analysis on the input data to identify one or more lesions, generating a probability map for the one or more lesions from the input data; performing a second analysis on the input data to obtain a malignancy probability mask for the input data; and generating an overlay for the input data by combining the lesion probability map with the malignancy probability mask.

FIELD

The present invention relates to deep learning for automated assessmentof malignancy of lesions.

BACKGROUND

Digital mammography is an X-ray system in which the X-ray film isreplaced with an electronic system that convert X-rays into images ofthe human breast. An important procedure prior to image analysis isimage segmentation, the slicing of an image into useful and meaningfulsegments for the ease of analysis. This procedure is often challenging,and the quality of the image may cause difficulty when it comes toaccurately and precisely detecting abnormalities or diseases. Thus,effecting the determination of diagnosis and therapeutic preparation.

Meaningful medical image data plays a key part in the stage of canceridentification and impacts patient treatment. In determining whether oneor more lesions are benign or malignant, medical professionals use asystem called classification, a method of descriptively assigningobjects into a predefined set of categories. This system allowsprofessionals to categorise lesions by possibility of malignancy.Although manual classification has reduced levels of miscommunicationand improved monitoring, drawbacks are yet present. Disadvantages ofsuch a traditional determination method include time-consumption, due tothe possibility of additional evaluations and manual screening, andpossible classification error, due to fault of the human eye, leading tofalse-positive and false-negative results, which may cause psychologicaland physiological effects.

SUMMARY OF INVENTION

Aspects and/or embodiments seek to provide a method, apparatus, andsystem for the automated assessment of malignancy of lesions.

According to a first aspect, there is provided a computer-aided methodof malignancy assessment of lesions, the method comprising the steps of:receiving input data; performing a first analysis on the input data toidentify one or more lesions, generating a probability map for the oneor more lesions from the input data; performing a second analysis on theinput data to obtain a malignancy probability mask for the input data;and generating an overlay for the input data by combining the lesionprobability map with the malignancy probability mask.

Conventional segmentation methods for identifying and assessing lesionsrely on an expert, usually a radiologist, providing a seed region orstarting point. Often, the radiologist will segment the entire imagewithout any computerised input, which can lead to errors owing tomistakes, carelessness, human error, and/or details too fine for thehuman eye to detect. Conversely, the method disclosed herein is operableto segment a region without any prior input other than an image.

Conventional methods rely on some form of initial seed information orfeature extraction. These methods are also interactive and require aradiologist to identify areas or perform particular tasks to begin with.In contrast, with this method a radiologist does not need to provide anyinputs to initiate or prepare for the analysis which therefore improvestime efficiency of the analysis. Additionally, by enabling the use of amalignancy model and a segmentation simultaneously, this methodeliminates or at least reduces the chances of false positives.

Optionally, the step of performing the first analysis on the input datais performed through a sliding window. Optionally, the step ofperforming the second analysis is performed substantially simultaneouslyto the step of performing the first analysis.

Optionally, the first analysis is performed using one or more FullyConvolutional Networks (FCNs). Optionally, the or each FCN comprises oneor more convolutional layers. Optionally, the or each FCN comprises oneor more hidden representations. Optionally, the or each FCN comprisesone or more activation layers, the one or more activation layerscomprising one or more rectified linear units (ReLU) and/or exponentiallinear units (ELU). Optionally, the or each FCN comprises one or moresigmoid activation layers and/or softmax functions for the or eachsegmented region.

Optionally, the second analysis is performed using one or moreConvolutional Neural Networks (CNNs). Optionally, the one or more CNNsare operable to identify distinguish between a malignant lesion and/or abenign lesion and/or typical tissue. Optionally, the one or more CNNsare operable to generate a malignancy model.

Convolutional networks are powerful tools inspired by biological neuralprocesses, which can be trained to yield hierarchies of features and areparticularly suited to image recognition.

Convolutional layers apply a convolutional operation to an input, andpass the results to a following layer. With training, FCNs and CNNs canachieve expert-level accuracy or greater with regard to segmenting andassessing anatomical and/or pathological regions and/or lesions indigital medical images such as mammograms.

Optionally, the overall prediction score is a mean score across aplurality of patches.

By applying a mean calculation across a plurality of patches, the numberof errors and/or inaccuracies may be reduced. An incorrect calculationfor one pixel in an overlapping area may be at least partially mitigatedby a correct calculation once the overlapping area is analysed again.

Optionally, the input data comprises medical image data. Optionally, themedical image data comprises one or more mammograms. Optionally, theinput data comprises one or more Digital Imaging and Communications inMedicine (DICOM) files.

FCNs can also analyse medical images far more quickly than a humanexpert, and hence increase the number of medical images analysedoverall. Therefore a problem, for example the growth of a canceroustumour, can be detected more quickly than waiting for a human expert tobecome available and hence treatment may begin earlier. Theidentification of regions of interest, which may include lesions, maytherefore aid screening and clinical assessment of breast cancer amongother medical issues. Earlier diagnosis and treatment can reducepsychological stress to a patient and also increase the chances ofsurvival in the long term.

Optionally, the output data comprises an overlay. Optionally, theoverlay comprises a segmentation outline and/or probability map showingone or more locations of one or more segmented regions.

Providing a clear and accurate segmentation of lesion regions can bevery helpful when reviewing a medical image, for example a mammogram.This may be especially relevant if there is reason to suspect there is amedical issue with a patient, for example a swollen area which is largerthan it was in previous scans. Such changes may be more easilydetectable if the different lesion regions are clearly segmented.

Optionally, voids within the segmentation outline are operable to beremoved. Optionally, one or more probability masks are generated for theone or more segmented regions. Optionally, one or more of the one ormore probability masks are converted to one or more binary masks.Optionally, the conversion of the one or more of the one or moreprobability masks to one or more binary masks is performed bythresholding the probabilities. Optionally, one or more parts of the oneor more binary masks are removed with reference to an assignedthreshold.

If a lesion region is detected and segmented, it is possible that aregion within that segmentation is incorrectly identified or notidentified at all. Therefore, there may be a void within the pectoralmuscle segmentation, or a different segmented region which is too smallto have been correctly identified as measured against a predeterminedthreshold. In order to correct this error that part of the segmentationmay be removed. For ease of further downstream analysis, the one or moreprobability masks may be in the form of one or more probability maps.The one or more binary masks may be in the form of one or more overlaysas described herein. The one or more binary masks may further compriseone or more quantized masks. The or any assigned threshold referred toherein may be established through trial and error, expert advice, and/ora tuning process performed before, during, and/or the training process.

Optionally, the one or more binary masks are upscaled to the originalsize of the input data. Optionally, the one or more binary masks arestored in the form of a DICOM file. Optionally, the one or more binarymasks comprise one or more identifications of masses and/orcalcifications. Optionally, the segmented regions comprise at least partof a human breast area.

As a DICOM file is conventionally used to store and share medicalimages, conforming to such a standard allows for easier distribution andfuture analysis of the medical images and/or any overlays or othercontributory data. The one or more binary masks may be stored as part ofa DICOM image file, added to an image file, and/or otherwise storedand/or represented according to the DICOM standard or portion of thestandard.

Lesions, which may comprise one or more cancerous growths, masses,abscesses, lacerations, calcifications, and/or other irregularitieswithin biological tissue, can cause serious medical problems if leftundetected. Such lesions are often conventionally detected and/oranalysed through a medical scan of a patient, which generates one ormore medical images such as a mammogram. Therefore, it is advantageousif such lesions are operable to be segmented, and hence reviewed withgreater accuracy by a medical professional. By also assessing theprobability of malignancy of lesions, appropriate medical action may betaken with the required urgency. Patients with benign lesions, or nolesions at all, may also be spared psychological pain from theuncertainty of knowing the danger that a lesion poses to their futurehealth.

According to a further aspect, there is provided an apparatus operableto perform the method disclosed herein. According to a further aspect,there is provided a system operable to perform the method disclosedherein.

Such an apparatus and/or system may be installed in or near hospitals,or connected to hospitals via a digital network, to reduce waiting timesfor medical images to be analysed. Patients may therefore be sparedstress from not knowing the results of a medical scan, and may receivetreatment more quickly if required. The apparatus and/or system and/ormethod disclosed herein may further form a constituent part of adifferent arrangement, for example detecting and/or segmenting differentobjects, environments, surroundings, and/or images.

According to a further aspect, there is provided a computer programproduct operable to perform the method and/or apparatus and/or system ofany preceding claim.

Through the use of a computer or other digital technology, segmentationof lesions from medical images and the assessment thereof may beperformed with greater accuracy, speed, and reliability that relying ona human expert. Therefore, a greater number of medical images may bereviewed, reducing backlogs for experts and further reducing errors madewhen the medical images themselves are actually reviewed.

According to a further aspect, there is provided a method of training aneural network to assess malignancy, the method comprises receivinginput data, performing a first analysis on the input data to identifyone or more lesions, generating a probability map for the one or morelesions from the input data, performing a second analysis on the inputdata to obtain a malignancy probability mask for the input data; andusing the lesion probability map and malignancy probability map to trainthe neural network.

BRIEF DESCRIPTION OF DRAWINGS

Embodiments will now be described, by way of example only and withreference to the accompanying drawings having like-reference numerals,in which:

FIG. 1 shows a training procedure for malignancy prediction;

FIG. 2 shows a run time procedure for malignancy prediction; and

FIG. 3 shows a more detailed example of a run time procedure formalignancy prediction.

SPECIFIC DESCRIPTION

Referring to FIG. 1, a first embodiment will now be described withreference to a training procedure. In this embodiment, a Digital Imagingand Communications in Medicine (DICOM) file is provided as input data101. DICOM is a conventional standard for transmitting and storingmedical images, for example a mammogram. Image data is then extractedfrom the DICOM file and an image is generated.

The image then undergoes a pre-processing stage 103. The pre-processingstage may comprise windowing the image data to a predetermined windowinglevel. The windowing level defines the range of bit values considered inthe image. Medical images are conventionally 16-bit images, wherein eachpixel is represented as a 16-bit integer ranging from 0 to 2¹⁶−1, i.e.[0, 1, 2, . . . , 65535]. The information content is very high in theseimages, and generally comprises more information than what the human eyeis capable of detecting. A set value for the window level is typicallyincluded within the DICOM file 102.

It can be important to maintain image resolution. In this embodimentconventional graphics processing unit (GPU) constraints require that theimage is divided into a plurality of patches in order to maintainresolution. Each patch is then provided to a Fully Convolutional Network(FCN). The larger the patch, the more context that can be provided butsome precision may be lost. For example, in the case of a large imagecomprising a small tumour, if the FCN is instructed that somewhere inthis patch there is a tumour, the network would need to learn how tofind it first before it can be classified. In this embodiment patchsizes of 300×300 pixels are used, although larger and smaller patchsizes may be used.

In order to ensure that a prediction for each patch corresponds to afinal output probability mask based on mean values for each pixel, asliding window arrangement is used. In order to use this arrangement,models may be trained on smaller patches sampled from an image. At“prediction/test time”, i.e. after the model has been trained, aprediction is required for every pixel of the image, but the input tothe model can only be a smaller patch owing to conventional hardwareconstraints. Therefore, the full image can be divided up into smallerpatches and fed individually into the FCN. The model “slides” over thefull images in a sliding window fashion and outputs a prediction foreach patch. The outputs are then stitched together to generate an outputmap. Therefore once training is complete, at prediction time the fullimage is divided into patches in the same sliding window fashion. Forexample, if each patch is 100×100 pixels every time we slide, we movethe patch with a specific number of pixels to the side (“the stride”).The second patch may comprise some overlap with a previous patch. Eachpatch is classified and that probability is given to every pixel withinthe patch. For example, if the probability of a patch being cancerous is0.9, then every pixel in that patch is labelled as 0.9. If there isoverlap, the mean of the number of overlapping pixels is calculated,although other arithmetic and/or mathematical operators may be used.

The FCN may comprise any combination of one or more convolutional,hidden representation, activation, and/or pooling layers. The activationlayer in this embodiment is in the form of a sigmoid activation layer.The FCN is trained to generate such probability mask by providing a setof input values and associated weights. The probability mask isgenerated taking the mean for every pixel from the patches to form afinal output probability mask 104.

During training of the FCN, a correct class for each value is known, andhence it is possible to compare the FCN's calculated output probabilitymask to the correct values. An error term or loss function for each nodein the FCN can then be established, and the weights adjusted, so thatfor future input values the output probability mask is closer to thecorrect value. Backpropagation techniques can be used in the trainingschedule for the or each neural network.

The model is trained using backpropagation and forward pass through thenetwork. The loss function for dense training is the sum over spatialdimensions of the loss functions of the individual pixels.

L(x)=Σ_(i,j) l′(x _(i,j))

here L(x) is the loss over the whole image and l′(x_(i,j)) is the lossfor the pixel at i, j. This enables the system to automatically identifyone or more tumours from the image created by the system.

The loss function may be the DICE loss, which is defined as

$L_{DSC} = \frac{2\Sigma_{i}^{N}s_{i}r_{i}}{{\Sigma_{i}^{N}s_{i}} + {\Sigma_{i}^{N}r_{i}}}$

where s_(i) and r_(i) represent the continuous values of the predictionmap ∈ [0, . . . , 1] and the ground truth at each pixel i, respectively.Alternatively, a cross-entropy can be used. The cross-entropy loss forthe pixel at i, j is defined as

$L_{CE} = {- {\sum\limits_{c = 1}^{c}{y*\log \; (s)}}}$

where C is the number of classes, y ∈ {0,1} is the binary indicator forclass c, and s is the score for class c. The loss for the full image, x,is defined as the sum over all the losses for the pixels:

${L_{CE}(x)} = {\sum\limits_{i,j}\left( {- {\sum\limits_{c = 1}^{c}{y*\log \; (s)}}} \right)}$

Once the probability mask has been generated, which in this embodimentmay be in the form of a probability map, one or more patches from theprobability map are sampled 105. The sampling may be proportional to theprobability of the presence of lesions, in particular the sampling maybe taken from areas with a higher probability of being a lesion asdefined by a predetermined threshold. Alternatively, Poisson sampling oruniform sampling may be used to sample patches from the probability map.Poisson sampling may give a better coverage of all of the breast tissue.The probability map in this embodiment is in the form of a tensor of thesame size as the input image, where each element is a probability ofbelonging to a class. A patch may comprise an entire probability map, ora portion of the probability map. Using one or more of these sampledpatches, a convolutional neural network (CNN) may be trained to generatea malignancy model 106, which in turn generates a malignancy mask. TheCNN may also be trained using the results of a different process, forexample a Random Forest based candidate selector or any similar lesiondetection method.

The malignancy mask can be generated through thresholding. For example,considering a malignancy model with three classes [“A”, “B”, “C”]. Apixel in the output tensor comprises three values [0.3, 0.3, 0.4].Therefore, this example would result in a probability vector for thatpixel of: 0.3 for class A, 0.3 for class B, and 0.4 for class C.

FIGS. 2 and 3 show an example of a run time procedure for malignancyprediction. This may also be referred to as prediction time and/or testtime. Once a CNN or other appropriate network has been trained as above,the malignancy of lesions may then be predicted with greater accuracy.One or more patches may be provided to the CNN through the use of asliding window analysis arrangement 302. As depicted in step 303, alesions segmentation model is applied to the image. The or eachprobability mask from the FCN is then converted to one or more binarymasks during a post-processing stage 304. The conversion from aprobability mask to binary mask may be through thresholding theprobabilities. Small areas in the binary mask may be removed. If thearea (which may be represented by an identified number of pixels) issmaller than a specific predetermined threshold, then the area may beremoved from the binary mask entirely. Similarly, holes in thesegmentation itself may be removed. If a segmentation has an area ofzeros, entirely surrounded by ones, then the zeros may be set to onesaccording to a predetermined threshold value for the area.

During a prediction, an image is analysed using both the CNN-generatedmalignancy mask and the FCN-generated probability map. During run time,the malignancy model and the lesion segmentation model process the imagesimultaneously. The probability map is then used to select one or morerelevant parts of the malignancy mask. Such a selection may be providedthrough multiplying the malignancy mask 305 with the one or more binarymasks 306. The result then undergoes a post-processing stage 307, duringwhich an overlay is generated. The overlay may comprise any markings oneor more parts of the original image, for example by outlining differentareas of human breast tissue, regions of interest, and/or marking one ormore levels of malignancy if a lesion is detected and can be stored inthe DICOM image 308.

The generation of the overlay is an entirely automated process, andrequires no human action other than the input of a data to be analysed.Conventional segmentation methods rely on an expert, usually aradiologist, providing a seed region, starting point or some form offeature engineering. Conversely, the method disclosed herein is operableto segment a region and hence assess malignancy without any prior inputother than an image.

Mammography is a medical imaging modality widely used for breast cancerdetection. Mammography makes use of “soft” X-rays to produce detailedimages of the internal structure of the human breast—these images arecalled mammograms and this method is considered to be the gold standardin early detection of breast abnormalities which provide a validdiagnosis of a cancer in a curable phase.

Unfortunately, the procedure of analysing mammograms is oftenchallenging. The density and tissue type of the breasts are highlyvaried and in turn present a high variety of visual features due topatient genetics. These background visual patterns can obscure the oftentiny signs of malignancies which may then be easily overlooked by thehuman eye. Thus, the analyses of mammograms often leads tofalse-positive or false-negative diagnostic results which may causemissed treatment (in the case of false negatives) as well as unwantedpsychological and sub-optimal downstream diagnostic and treatmentconsequences (in the case of false positives).

Most developed countries maintain a population-wide screening program,comprising a comprehensive system for calling in women of a certain agegroup (even if free of symptoms) to have regular breast screening. Thesescreening programs require highly standardized protocols to be followedby experienced specialist trained doctors who can reliably analyse alarge number of mammograms routinely. Most professional guidelinesstrongly suggest reading of each mammogram by two equally expertradiologists (also referred to as double-reading). Nowadays, when thenumber of available radiologists is insufficient and decreasing, thedouble-reading requirement is often impractical or impossible.

When analysing mammograms, the reliable identification of lesionstructures is important for visual evaluation and especially foranalytic assessment of visual features based on their anatomic locationand their relation to anatomic structures, which may have profoundimplications on the final diagnostic results. In the case that anatomicstructures appear distorted they may also indicate the presence ofpossible malignancies.

Conventional X-ray is a medical imaging modality widely used for thedetection of structural abnormalities related to the air containingstructures and bones, as well as those diseases which have an impact onthem. Conventional X-ray is the most widely used imaging method andmakes use of “hard” X-rays to produce detailed images of the internalstructure of the lungs and the skeleton. These images are calledroentgenograms or simply X-rays.

Unfortunately, the procedure of analysing X-rays is often challenging,especially when analysing lung X-rays in order to detect infectiousdisease (e.g. TB) or lung cancer in early stage.

Most developed countries maintain a population-wide screening program,comprising a comprehensive system for calling in the population of acertain age group (even if free of symptoms) to have regular chest X-rayscreening. These screening programs require highly standardizedprotocols to be followed by experienced specialist trained doctors whocan reliably analyse a large number of X-rays routinely.

When analysing X-ray images, the reliable identification of lesionstructures is important for visual evaluation and especially foranalytic assessment of visual features based on their anatomic locationand their relation to anatomic structures, which may have profoundimplications on the final diagnostic results. In the case that anatomicstructures appear distorted they may also indicate the presence ofpossible malignancies.

Cross-sectional medical imaging modalities are widely used for detectionof structural or functional abnormalities and diseases which have avisually identifiable structural impact on the human internal organs.Generally the images demonstrate the internal structures in multiplecross-sections of the body. The essence of the most widely usedcross-sectional techniques are described below.

Computed tomography (CT) is a widely used imaging method and makes useof “hard” X-rays produced and detected by a specially rotatinginstrument and the resulted attenuation data (also referred to as rawdata) are presented by a computed analytic software producing detailedimages of the internal structure of the internal organs. The producedsets of images are called CT-scans which may constitute multiple serieswith different settings and different contrast agent phases to presentthe internal anatomical structures in cross sections perpendicular tothe axis of the human body (or synthesized sections in other angles).

Magnetic Resonance Imaging (MRI) is an advanced diagnostic techniquewhich makes use of the effect magnetic field impacts on movements ofprotons which are the utmost tiniest essential elements of every livingtissue. In MRI machines the detectors are antennas and the signals areanalysed by a computer creating detailed images if the internalstructures in any section of the human body. MRI can add usefulfunctional information based on signal intensity of generated by themoving protons.

However, the procedure of analysing any kind of cross-sectional imagesis often challenging, especially in the case of oncologic disease as theinitial signs are often hidden and appearance of the affected areas areonly minimally differed from the normal.

When analysing cross sectional scans, diagnosis is based on visualevaluation of anatomical structures. The reliable assessment, especiallyfor analytic assessment, of visual appearance based on their anatomiclocation and their relation to anatomic structures, may have profoundimplications on final diagnostic results. In the case that anatomicstructures appear distorted they may also indicate the presence ofpossible malignancies.

Generally, in the case of all diagnostic radiology methods (whichinclude mammography, conventional X-ray, CT, MRI), the identification,localisation (registration), segmentation and classification ofabnormalities and/or findings are important interlinked steps in thediagnostic workflow.

In the case of ordinary diagnostic workflows carried out by humanradiologists, these steps may only be partially or sub-consciouslyperformed but in the case of computer-based or computer-aided diagnosesand analyses the steps often need to be performed in a clear, concrete,descriptive and accurate manner.

Locality and classification may define and significantly influencediagnoses. Both locality and classification may be informed bysegmentation in terms of the exact shape and extent of visual features(i.e. size and location of boundaries, distance from and relation toother features and/or anatomy). Segmentation may also provide importantinformation regarding the change in status of disease (e.g. progressionor recession).

Machine learning is the field of study where a computer or computerslearn to perform classes of tasks using the feedback generated from theexperience or data gathered that the machine learning process acquiresduring computer performance of those tasks.

Typically, machine learning can be broadly classed as supervised andunsupervised approaches, although there are particular approaches suchas reinforcement learning and semi-supervised learning which havespecial rules, techniques and/or approaches. Supervised machine learningis concerned with a computer learning one or more rules or functions tomap between example inputs and desired outputs as predetermined by anoperator or programmer, usually where a data set containing the inputsis labelled.

Unsupervised learning is concerned with determining a structure forinput data, for example when performing pattern recognition, andtypically uses unlabelled data sets. Reinforcement learning is concernedwith enabling a computer or computers to interact with a dynamicenvironment, for example when playing a game or driving a vehicle.

Various hybrids of these categories are possible, such as“semi-supervised” machine learning where a training data set has onlybeen partially labelled. For unsupervised machine learning, there is arange of possible applications such as, for example, the application ofcomputer vision techniques to image processing or video enhancement.Unsupervised machine learning is typically applied to solve problemswhere an unknown data structure might be present in the data. As thedata is unlabelled, the machine learning process is required to operateto identify implicit relationships between the data for example byderiving a clustering metric based on internally derived information.For example, an unsupervised learning technique can be used to reducethe dimensionality of a data set and attempt to identify and modelrelationships between clusters in the data set, and can for examplegenerate measures of cluster membership or identify hubs or nodes in orbetween clusters (for example using a technique referred to as weightedcorrelation network analysis, which can be applied to high-dimensionaldata sets, or using k-means clustering to cluster data by a measure ofthe Euclidean distance between each datum).

Semi-supervised learning is typically applied to solve problems wherethere is a partially labelled data set, for example where only a subsetof the data is labelled. Semi-supervised machine learning makes use ofexternally provided labels and objective functions as well as anyimplicit data relationships. When initially configuring a machinelearning system, particularly when using a supervised machine learningapproach, the machine learning algorithm can be provided with sometraining data or a set of training examples, in which each example istypically a pair of an input signal/vector and a desired output value,label (or classification) or signal. The machine learning algorithmanalyses the training data and produces a generalised function that canbe used with unseen data sets to produce desired output values orsignals for the unseen input vectors/signals. The user needs to decidewhat type of data is to be used as the training data, and to prepare arepresentative real-world set of data. The user must however take careto ensure that the training data contains enough information toaccurately predict desired output values without providing too manyfeatures (which can result in too many dimensions being considered bythe machine learning process during training, and could also mean thatthe machine learning process does not converge to good solutions for allor specific examples). The user must also determine the desiredstructure of the learned or generalised function, for example whether touse support vector machines or decision trees.

The use of unsupervised or semi-supervised machine learning approachesare sometimes used when labelled data is not readily available, or wherethe system generates new labelled data from unknown data given someinitial seed labels.

Machine learning may be performed through the use of one or more of: anon-linear hierarchical algorithm; neural network; convolutional neuralnetwork; recurrent neural network; long short-term memory network;multi-dimensional convolutional network; a memory network; fullyconvolutional network or a gated recurrent network allows a flexibleapproach when generating the predicted block of visual data. The use ofan algorithm with a memory unit such as a long short-term memory network(LSTM), a memory network or a gated recurrent network can keep the stateof the predicted blocks from motion compensation processes performed onthe same original input frame. The use of these networks can improvecomputational efficiency and also improve temporal consistency in themotion compensation process across a number of frames, as the algorithmmaintains some sort of state or memory of the changes in motion. Thiscan additionally result in a reduction of error rates.

Developing a machine learning system typically consists of two stages:(1) training and (2) production. During the training the parameters ofthe machine learning model are iteratively changed to optimise aparticular learning objective, known as the objective function or theloss. Once the model is trained, it can be used in production, where themodel takes in an input and produces an output using the trainedparameters.

Any system feature as described herein may also be provided as a methodfeature, and vice versa. As used herein, means plus function featuresmay be expressed alternatively in terms of their correspondingstructure.

Any feature in one aspect of the invention may be applied to otheraspects of the invention, in any appropriate combination. In particular,method aspects may be applied to system aspects, and vice versa.Furthermore, any, some and/or all features in one aspect can be appliedto any, some and/or all features in any other aspect, in any appropriatecombination.

It should also be appreciated that particular combinations of thevarious features described and defined in any aspects of the inventioncan be implemented and/or supplied and/or used independently.

1. A computer-aided method of malignancy assessment of lesions, themethod comprising: receiving input data; performing a first analysis onthe input data to identify one or more lesions; generating a probabilitymap for the one or more lesions from the input data; performing a secondanalysis on the input data to obtain a malignancy probability mask forthe input data; and generating an overlay for the input data bycombining the lesion probability map with the malignancy probabilitymask.
 2. The method of claim 1, wherein performing the first analysis onthe input data is performed through a sliding window.
 3. The method ofclaim 1, wherein performing the second analysis is performedsubstantially simultaneously to the step of performing the firstanalysis.
 4. The method of claim 1, wherein the first analysis isperformed using one or more Fully Convolutional Networks (FCNs), and/orthe second analysis is performed using one or more Convolutional NeuralNetworks (CNNs).
 5. The method of claim 4, wherein the or each FCNcomprises one or more convolutional layers and/or one or more hiddenrepresentations.
 6. (canceled)
 7. The method of claim 4, wherein the oreach FCN comprises one or more activation layers, the one or moreactivation layers comprising one or more rectified linear units (ReLU)and/or exponential linear units (ELU).
 8. The method of claim 4, whereinthe or each FCN comprises one or more sigmoid activation layers and/orsoftmax functions for each of one or more segmented regions. 9.(canceled)
 10. The method of claim 9, wherein the one or more CNNs areoperable to identify distinguish between a malignant lesion and/or abenign lesion and/or typical tissue.
 11. The method of claim 4, whereinthe one or more CNNs are operable to generate a malignancy model. 12.The method of claim 1, wherein the input data further comprises one ormore patches, the method further comprising: calculating an overallprediction score for the or each patches; and determining an overallprediction score which is a mean score across a plurality of patches.13. The method of claim 1, wherein the input data comprises medicalimage data and/or one or more Digital Imaging and Communications inMedicine (DICOM) files.
 14. The method as claimed in claim 13, whereinthe medical image data comprises one or more mammograms.
 15. (canceled)16. The method of claim 1, wherein the overlay comprises a selection ofone or more elements of the malignancy probability mask based on anapplication of the probability map.
 17. The method of claim 16, whereinthe overlay comprises a segmentation outline and/or probability mapshowing one or more locations of one or more segmented regions.
 18. Themethod of claim 17, wherein voids within the segmentation outline areoperable to be removed.
 19. (canceled)
 20. The method of claim 17,further comprising one or more of: generating one or more probabilitymasks for the one or more segmented regions; converting one or more ofthe one or more probability masks to one or more binary masks, whereinthe one or more binary masks comprise one or more identifications ofmasses and/or calcifications, wherein the converting is performed bythresholding the probabilities; and/or removing one or more parts of theone or more binary masks with reference to an assigned threshold. 21.(canceled)
 22. (canceled)
 23. The method of claim 20, wherein the one ormore binary masks are one or both upscaled to the original size of theinput data and/or stored in the form of a DICOM file.
 24. (canceled) 25.(canceled)
 26. (canceled)
 27. An apparatus operable to perform themethod of claim
 1. 28. (canceled)
 29. A computer program productincluding one or more non-transitory machine readable mediums encodedwith instructions that when executed by one or more processors cause aprocess to be carried out for assessing malignancy of lesions, theprocess comprising receiving input data; performing a first analysis onthe input data to identify one or more lesions; generating a probabilitymap for the one or more lesions from the input data; performing a secondanalysis on the input data to obtain a malignancy probability mask forthe input data; and generating an overlay for the input data bycombining the lesion probability map with the malignancy probabilitymask.
 30. A method of training a neural network to assess malignancy,the method comprising: receiving input data; performing a first analysison the input data to identify one or more lesions; generating aprobability map for the one or more lesions from the input data;performing a second analysis on the input data to obtain a malignancyprobability mask for the input data; and using the lesion probabilitymap and malignancy probability map to train the neural network.